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Abstract 

Background: Every two years, long-term care organizations for the elderly are obliged to evaluate and publish the 
experiences of residents, representatives of psychogeriatric patients, and/or assisted-living clients with regard to quality 
of care. Our hypotheses are that publication of this quality information leads to improved performance, and that 
organizations with substandard performance will improve more than those whose performance is relatively good. 

Methods: The analyses included organizational units that measured experiences twice between 2007 (t 0 ) and 2009 
(t]). Experiences with quality of care were measured with Consumer Quality Index (CQI) questionnaires. Besides 
descriptive analyses (i.e. mean, 5 th and 95 th percentile, and 90% central range) of the 19 CQI indicators and change 
scores of these indicators were calculated. Differences across five performance groups (ranging from 'worst' to 
'best') were tested using an ANOVA test and effect sizes were measured with omega squared (co 2 ). 

Results: At t 0 experiences of residents, representatives, and assisted-living clients were positive on all indicators. 
Nevertheless, most CQI indicators had improved scores (up to 0.37 change score) at t h Only three indicators showed a 
minor decline (up to -0.08 change score). Change scores varied between indicators and questionnaires, e.g. they were 
more profound for the face-to-face interview questionnaire for residents in nursing homes than for the other two mail 
questionnaires (0.15 vs. 0.05 and 0.04, respectively), possibly due to more variation between nursing homes on the first 
measurement, perhaps indicating more potential for improvement. A negative relationship was found between prior 
performance and change, particularly with respect to the experiences of residents (co 2 = 0.16) and assisted-living clients 
(co 2 = 0.15). However, the relation between prior performance and improvement could also be demonstrated with 
respect to the experiences reported by representatives of psychogeriatric patients and by assisted-living clients. For 
representatives of psychogeriatric patients, the performance groups 1 and 2 ([much] below average) improved 
significantly more than the other three groups (co 2 = 0.05). 

Conclusions: Both hypotheses were confirmed: almost all indicator scores improved over time and long-term care 
organizations for the elderly with substandard performance improved more than those with a performance which 
was already relatively good. 



Background 

Various countries publish quality information about 
nursing homes and residential care facilities on the 
Internet in the form of report cards which serve multi- 
ple purposes [1-3]. First, quality information is available 
for choices by (future) clients or for families of these 



* Correspondence: m.zuidgeest@casema.nl 

1 TRANZO, Scientific Centre for care and welfare, Faculty of Social and 
Behavioral Sciences, Tilburg University, PO Box 90153, 5000 LE Tilburg, the 
Netherlands 

Full list of author information is available at the end of the article 



clients. Second, it can be used by nursing homes to 
account for their performance to healthcare regulators 
and government. Third, quality information informs 
health insurers about performance differences between 
nursing homes, which can be incorporated in their pur- 
chasing decisions. Finally, quality information can be 
used by nursing homes themselves to monitor and 
improve their quality of care [4]. In this article, the 
focus is on the latter function of public report cards, i.e. 
the role of transparency in quality improvement. 
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Countries differ in the quality information they pro- 
vide on the report cards. Quality information can 
include structure indicators which refers to the condi- 
tions under which care is provided, process indicators 
which relates to the professional activities associated 
with providing care, and outcome indicators which 
denotes the effects of care [5]. Most countries provide 
the first two types of indicators and only some countries 
(Austria, Germany, the Netherlands, Sweden, USA) pre- 
sent the outcome measurements such as care-related 
safety, satisfaction, and experiences of residents or 
representatives [2]. 

In the Netherlands, the national indicator set for long- 
term care for the elderly is called the 'Quality Frame- 
work Responsible Care' [4]. It includes patient experience 
indicators which are measured with three separate ques- 
tionnaires: a face-to-face interview protocol for residents 
in nursing homes, a mail questionnaire for representa- 
tives of psychogeriatric patients in residential care facil- 
ities, and a mail questionnaire for assisted-living clients 
that receive care from home care organizations. These 
questionnaires belong to the so-called Consumer Qual- 
ity Index (CQI), which is the Dutch standard for mea- 
suring patient and client experiences in healthcare [6]. 
The national indicator set also includes clinical indica- 
tors like skin problems, depression, fall incidents, physi- 
cal restraints, malnutrition, and medication errors. The 
present study focuses on indicators of patient 
experience. 

Every organizational unit in long-term care (mostly a 
location of a nursing home, residential care facility or 
home care organization) is obliged to measure their per- 
formance on patient experience and clinical indicators. 
This obligation is defined by a national steering commit- 
tee which includes representatives from client organiza- 
tions, providers of long-term care for the elderly, 
healthcare regulators, the Dutch Ministry of Health, 
Welfare and Sport, and health insurers [7]. This national 
agreement is reinforced by the umbrella organization of 
nursing homes, homes for the elderly, and home care 
organizations, called ActiZ. Membership of ActiZ is ter- 
minated if members do not comply with the obligation 
to measure and publish client experiences. In addition, 
health insurers will cut the budget of healthcare provi- 
ders who do not measure these indicators. These incen- 
tives spur care homes to measure and publish these 
scores. In 2007 (at the start of publishing the results), 
62% of the organizational units published their out- 
comes [8] which increased to nearly all homes in 2010. 

Providers can choose their own time at which they 
measure the experiences of residents, representatives 
and assisted-living clients, as long as they conduct the 
CQI surveys with an interval of about two years. Since 
2007, several providers have measured and published 



their CQI survey results twice. In the present study we 
analyse CQI survey results of those providers in order 
to describe changes in their performance over time. 
Analyses of performance over time are made while tak- 
ing into account all three CQI instruments for long- 
term care of the elderly. The results of those surveys are 
published on the level of indicators. Indicators corre- 
spond to the items and scales that were established dur- 
ing development of the questionnaires [9]. 

We hypothesize that publication of the CQI survey 
results of the first measurement (t 0 ) will trigger activities 
to improve quality, which will lead to improved perfor- 
mance on the second measurement (ti). For example, in 
the USA care homes reorganized quality improvement 
programs and started new quality-assurance programs 
[10-12]. In addition, homes with poor quality scores 
were more likely to act on these performance scores 
than homes with better scores [10], a phenomenon that 
has also been observed in hospitals [13] and for health 
plans [14]. Thus, we hypothesize that organizations with 
substandard performance on the first measurement will 
show more improvement than organizations whose per- 
formance was already relatively good. 

Our first research question is 'Have scores with respect 
to experiences of residents in nursing homes, representa- 
tives of psychogeriatric patients in residential care facil- 
ities, and assisted-living clients receiving care from home 
care providers improved between the first (t 0 ) and the 
second (tj) measurement?' Furthermore, investigation of 
scores takes place with respect to nursing homes, resi- 
dential care facilities, and home care organizations that 
performed '(much) below average' on the first measure- 
ment compared with organizations that performed on 
average' and '(much) above average'. Therefore, the sec- 
ond research question is: 'Have nursing homes, residen- 
tial care facilities, and home care providers that 
performed '(much) below average' improved more 
between t 0 and tj than those that performed 'average' 
and '(much) above average'? 

Methods 

Sample 

In the Netherlands, long-term care is generally provided 
at home or in nursing homes or residential homes 
(either in somatic or psychogeriatric wards or care 
units). These providers publish their CQI indicator 
scores in their Annual Reports (http://www.jaarversla- 
genzorg.nl). The dataset of indicator scores is publicly 
available. Between 2007 and 2009 a total of 499 organi- 
zational units (mostly a location of a nursing home, resi- 
dential care facility or home care organization) had 
performed CQI surveys twice. Of these organizational 
units, 370 published CQI findings (at t 0 and ti) of the 
face-to-face interviews with residents, 190 published 
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findings of the mail questionnaire for representatives of 
psychogeriatric patients, and 122 published findings of 
the mail questionnaire for clients receiving care at home 
(assisted-living) questionnaires. One organizational unit 
can perform multiple questionnaires. 

Quality Framework Responsible Care 

Client and clinical indicators belong to the Quality Fra- 
mework 'Responsible Care' which encompasses seven 
quality domains. Four domains relate to quality of life: 
physical well-being and health, domestic and living con- 
ditions, participation and social handiness, mental well- 
being. Other domains are quality of caregivers, quality 
of care organization, and indicators with respect to 
more technical aspects of care (Table 1) [15]. 

CQI questionnaires and client indicators 

Every two years, the actual experiences of residents are 
measured in a face-to-face interview, whereas experi- 
ences of family members and assisted-living clients are 
measured with mail questionnaires. Typically, response 
categories in the interview protocol and mail question- 
naires refer to the frequency with which quality criteria 
were met: T never, '2' sometimes, '3' usually, and '4' 
always. The questionnaires belong to the CQI Long- 
term Care [9]. Collecting CQI data (e.g. conducting 
interviews, sending postal questionnaires and sending 
reminders), analysing and reporting the results is not 
done by the organizational units themselves, but can 
only be performed by certified survey vendors. Since 
this process takes considerable time and effort, data col- 
lection is possible during a two-year period. Data of all 
organizations were stored in a national databank. 

CQI indicators 

The development of the CQI surveys and indicators is 
described elsewhere [16-18], including a report specifi- 
cally focused on the CQI Long-term Care [9]. In brief, 
CQI indicators were determined by factor and reliability 
analysis [9,19]. The scores of the indicators were calcu- 
lated for each respondent provided that half or more of 
the items were available. All indicator scores ranged 
from 1 to 4, where a score of '4' represents the 'best' 
rating. In total, 19 indicators (items and scales) were 
distinguished with the three questionnaires (Table 1). 
The analytical strategy used to compare CQI scores 
between healthcare providers is described elsewhere 
[20-22]. In brief, multi-level linear regression analyses 
(respondents were nested within homes) were per- 
formed to yield an empirical Bayes (EB) estimate per 
indicator and per organizational unit. The EB estimates 
an organization's mean indicator score, based on the 
scores within that particular organization and the distri- 
bution of the scores of all the other organizations. This 



method is widely recommended for analysis of institu- 
tional performance [22-24] and results in more stable 
estimates. In order to compare the EB estimates of orga- 
nizations' indicator scores, so-called comparison inter- 
vals (calculated as ± 1.39 * SE) are used instead of 95% 
confidence intervals. These comparison intervals ensure 
that non-overlapping intervals of EB estimates represent 
a significant difference in indicator scores (p < 0.05) 
[25]. Apart from that, indicator scores were corrected 
for case-mix because client populations may differ on 
characteristics beyond the control of care providers 
[22,26]. For the interview protocol, the case-mix vari- 
ables were age, education, perceived health, and length 
of stay. The indicator scores of the mail questionnaire 
to representatives were corrected for the kind of repre- 
sentative (e.g. spouse, son or daughter), residents' age 
and education, and length of stay. For the assisted-living 
clients questionnaire indicator scores were corrected for 
age, education, length of care, help with completing the 
questionnaire, and kind of care (cleaning house, perso- 
nal care or assistant) [26]. The corrected indicator 
scores were divided into five performance groups after 
examining the data using: 1) the average score of all 
organizations on one indicator, 2) the average score of 
the higher bound of the comparison intervals, and 3) 
the average score of the lower bound of the comparison 
intervals. Figure 1 shows the classification of the perfor- 
mance groups representing '1' performance much below 
average (worst) to '5' representing performance much 
above average (best). We performed secondary analyses 
based on the corrected scores on two measurement 
points. 

Analysis 

Per questionnaire, descriptive analyses on indicators 
scores (mean scores, the observed 5 th and 95 th percen- 
tile, and the 90% central range) were performed on the 
data of the first measurement (t 0 ). The 90% central 
range is defined by the 5th and 95th percentiles to pro- 
vide a stable measure of observed dispersion. This is an 
indication of potential for improvement, because greater 
dispersion means that lower performing organizations 
can make more improvement, as there will be a larger 
difference with higher performing organizations. A 
change score per indicator was calculated by subtracting 
the indicator scores between those two measurement 
points (t 0 - ti). Per CQI questionnaire and per indicator 
we analysed these change scores. In addition, we exam- 
ined the overall change scores per questionnaire for dif- 
ferent so-called performance groups. Performance 
groups refer to the performance scored at t 0 and ranged 
from 1 (much below average) to 5 (much above average) 
(Figure 1). Differences in change scores across perfor- 
mance groups were tested using an ANOVA test. If the 
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Table 1 Seven themes and patient experience indicator scores (mean, 5 and 95 percentile, and the 90% central 
range) within the Quality Framework Responsible Care of the first measurement (t 0 ) 



Questionnaire 




Residents 








Represen- 
tatives 








Assisted-living 
clients 








Indicator 


Mean 


P05 


P95 


90% 
range 


Mean 


P05 


P95 


90% 
range 


Mean 


P05 


P95 


90% 
range 


1 Physical well-being and 
health 


1_1 Body care 


3.38 


3.08 


3.60 


0.52 


3.15 


2.93 


3.31 


0.38 


3.46 


3.31 


3.59 


0.28 


1_2 Meals 


2.94 


2.44 


3.36 


0.92 


3.45 


3.26 


3.59 


0.33 










2 Domestic and living 
conditions 


2_1 Cleaning 


3.28 


2.75 


3.63 


0.88 


3.14 


2.70 


3.49 


0.79 










2_2 Atmosphere 


3.36 


3.00 


3.61 


0.61 


3.07 


2.77 


3.27 


0.50 










2_3 Housing and privacy 


3.69 


3.10 


3.94 


0.84 


3.47 


2.67 


3.92 


1.25 










2_4 Experience of safety 


3.72 


3.55 


3.83 


0.28 


2.80 


2.5 


3.05 


0.55 


3.43 


3.26 


3.55 


0.29 


3 Participation and social 
handiness 


3_1 Daily activities/ 


3.41 


3.11 


3.64 


0.53 


2.92 


2.59 


3.16 


0.57 


2.84 


2.62 


3.00 


0.38 


participation 


























3_2 Autonomy 


3.34 


2.64 


3.72 


1.08 










3.42 


3.35 


3.48 


0.13 


4 Mental well-being 


4_1 Mental well-being 


3.18 


2.96 


3.37 


0.41 


3.22 


3.01 


3.39 


0.38 


3.37 


3.30 


3.45 


0.15 


5 Quality of caregivers 


5_1 Professionalism + safety 


3.43 


3.14 


3.67 


0.53 


3.28 


3.09 


3.4 


0.31 


3.51 


3.35 


3.63 


0.28 


care giving 


























5_2 Respectful treatment 


3.37 


3.06 


3.62 


0.56 


3.46 


3.29 


3.58 


0.29 


3.58 


3.42 


3.67 


0.25 


5_3 Reliability caregivers 










3.15 


2.88 


3.40 


0.52 


3.67 


3.56 


3.74 


0.18 


6 Quality of care 
organization 


6_1 Care plan + evaluation 


3.16 


2.60 


3.66 


1.06 


3.41 


3.19 


3.63 


0.44 


3.65 


3.56 


3.71 


0.15 


6_2 Shared decision making 


2.61 


2.09 


3.14 


1.05 


2.81 


2.59 


3.00 


0.41 


2.94 


2.76 


3.13 


0.37 


6_3 Information 


2.74 


2.23 


3.17 


0.94 


3.25 


2.88 


3.56 


0.68 


3.16 


2.99 


3.32 


0.33 


6_4 Telephone accessibility 










3.35 


3.16 


3.48 


0.32 


3.23 


2.94 


3.46 


0.52 


6_5 Coherence in care 


















3.08 


2.80 


3.34 


0.54 


6_6 Availability staff 


2.92 


2.51 


3.26 


0.75 


2.97 


2.66 


3.17 


0.51 


3.25 


2.97 


3.41 


0.44 


7 Technical aspects 


7_12 Physical restraints 










3.48 


3.16 


3.65 


0.49 










Average score per 


3.23 


2.82 


3.55 


0.73 


3.19 


2.90 


3.41 


0.51 


3.33 


3.16 


3.46 


0.30 


questionnaire 



























assumption of equal variances was violated, the Welch 
test of robust test of equality of means was used. Tukey 
post-hoc tests were used to compare the mean differ- 
ence score of every performance group to the means of 
every other performance group, and identifies where the 
differences between two means is greater than the stan- 
dard error would be expected. Effect sizes were mea- 
sured with omega squared (go 2 ) [25]. Omega squared 
estimates the proportion of variance explained for the 
population and is calculated as ((SS Be tween-(dfBetween 



*MS Within ))/(SS Tota i+MS w 

ithin)' A small, medium and 
large effect are defined as co 2 = 0.01, 0.06 and 0.14, 
respectively [27]. A p-value < 0.05 was considered statis- 
tically significant. Analyses were performed using SPSS 
version 17.0. 

Results 

Table 1 presents per indicator and per questionnaire the 
mean scores, the 5 th and 95 th percentile, and the 90% 
central range. At t 0 the experiences of respondents on 
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Upper bound 95% CI 

Average score of all 
homes on one indicator 

Low bound 95% CI 



I High and low bound for 
[ 95% CI 

• Average indicator score 



Performance group 1 2 3 4 5 

Figure 1 Based on the indicator score and the 95% confidence interval, determination of the performance groups 1 ('worst 7 ) to 5 
('best'). 



all indicators were positive for the interview question- 
naire, as were the experiences of representatives and 
assisted-living clients on the mail questionnaire: 3.23 
(2.82-3.55), 3.19 (2.90-3.41), and 3.33 (3.16-3.46), respec- 
tively. For the interview questionnaire the 90% range 
was 0.73, and for the mail questionnaires (representa- 
tives and assisted-living clients) it was 0.51 and 0.30, 
respectively. 

The largest difference between the 5 th and 95 th per- 
centile of the interview questionnaire was for the indica- 
tors 3_2 'Autonomy' (1.08), 6_1 'Care plan and 
evaluation' (1.06), and 6_2 'Shared decision making' 
(1.05). Housing and privacy (indicator 2_3) is an indica- 
tor of the representatives' questionnaire that showed 
considerable variation between organizational units 
(1.25). No indicators of the third questionnaire diverged 
as much as the other questionnaires. Indicator 6_5 
'Coherence in care' showed the most variation (0.54). 

Figure 2 shows the change scores per indicator for all 
three questionnaires. We expected change scores to be 
limited by the 90% observed range on the first measure- 
ment; this range varied between indicators and ques- 
tionnaires (Table 1). Theoretically, an indicator score 
can change 3 points (from 1 to 4, and vice versa). The 
change scores for the interview questionnaires were all 
positive (0.06-0.37). For the interviews with residents 
and the mail questionnaire for representatives, the three 
indicators that improved the most were the indicators 
6_2 'Shared decision making' (0.37; 0.23), 6_1 'Care plan 
and evaluation' (0.36; 0.15), and 6_3 'Information' (0.26; 
0.10). Indicator 2_3 'Housing and privacy' also changed 
by 0.10 for the representatives questionnaire. Scores of 
the questionnaire for assisted-living clients improved the 
most with respect to indicators 6_2 'Shared decision 
making' (0.25), 6_5 'Coherence in care' (0.09), and 6_4 
'Telephone accessibility'(0.08). The surveys of the repre- 
sentatives showed a decline in indicator scores 4_1 
'Mental well-being' (-0.04), 6_6 'Availability staff (-0.02), 



and 2_2 'Atmosphere' (-0.01). In the assisted-living cli- 
ents questionnaire, three indicators showed a decline: 
indicators 6_1 'Care plan and evaluation' (-0.08), 3_1 
'Daily activities' (-0.05), and 6_6 'Availability staff 
(-0.05). The change score of the assisted-living clients 
(range -0.08 to 0.25) did not diverge to the same extent 
as the change scores of the other questionnaires. 

To illustrate the differences in scores between ques- 
tionnaires, Figure 3 shows the mean change scores over 
all indicators between the two measurement points (t 0 
and ti). As expected, there was a negative relationship 
between prior performance and change (Figure 3). At ti 
the organizational units of performance groups 1, 2 and 
3 showed an average improvement of 0.34 (SD = 0.29), 
0.27 (0.23), and 0.11 (0.20), respectively. At t x the orga- 
nizational units of performance groups 4 and 5 showed 
an average improvement of 0.06 (0.12) and 0.02 (0.12), 
respectively. 

Change scores of the indicators based on the interview 
questionnaire changed to a greater extent than those of 
the other two questionnaires: all scores between the per- 
formance groups showed a significant difference (p < 
0.001) and the effect was large (oo 2 = 0.16). Regarding 
the questionnaire for representatives there was a signifi- 
cant difference between the performance groups (p < 
0.001), which was a medium effect (<d 2 = 0.05). Perfor- 
mance groups 1 and 2 differed from all the other perfor- 
mance groups. Performance groups 3, 4 and 5 did not 
differ from each other (p > 0.05). For the questionnaire 
for assisted-living clients there was a significant differ- 
ence between all four performance groups (p < 0.001), 
which was a large effect (oo 2 = 0.15). Performance group 
5 was excluded from the analysis, because there was 
only one indicator in group 5 at t 0 . 

Combining change scores with the scores at the first 
measurement (t 0 ) provides insight into the improvement 
potential per indicator. To illustrate this, Figure 4 shows 
the scores at t 0 together with the change scores (ti - t 0 ) 
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Questionnaire 

■ Residents 

□ Representatives 

□ Assisted-living clients 
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|f| 
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cr o " !-. 



5 



indicator 



Figure 2 Change scores of all indicators and the 95% confidence interval 



for the indicators based on the interview questionnaire. 
The same information for the indicators can be con- 
structed for the other two questionnaires (presented in 
Additional file 1), but the main findings of both figures 



are described. The three highest indicator scores at the 
second measurement (t x ) were for indicators 2_4 
'Experience of safety' (3.78), 2_3 'Housing and privacy' 
(3.76) and 5_1 'Professionalism and safety care giving' 



o 
c 

03 
-C 

O 



C 0,40" 
CM 
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P3 

W 

^ 0,30" 

o 
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I 
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0,10" 
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Questionnaire 

■ Residents 

□ Representatives 

□ Assisted-living clients 



Performance group on first measurement 
Figure 3 Mean difference in average score on all indicators and the 95% confidence interval. 
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2.4 Experience of safety 
2.3 Housing and privacy 
3.1 Professionalism and safety care giving 
6.1 Care plan and evaluation 
5.2 Respectful treatment 
3.1 Daily activities and participation 

2.2 Atmosphere 
3.2 Autonomy 
1.1 Body care 

2.1 Cleaning 
4.1 Mental well-being 
6.6 Availability staff 
1.2 Meals 

6.3 Information 
6.2 Shared decision making 




3.72 
3.69 



3.43 



2.60 2.80 3.00 3.20 

first measurement ■ change score 



3.40 



3.60 



3.80 



Figure 4 Per indicator, the average score of the first measurement (t 0 ) and the change score (t-, 
questionnaire for residents. 



t 0 ) for the face-to-face interview 



(3.55). The three highest scores of the questionnaire for 
representatives were for the indicators 2_3 'Housing and 
privacy' (3.58), 6_1 'Care plan and evaluating' (3.56), and 
7_12 'Physical restrains' (3.51). Finally, the three highest 
scores for the assisted-living clients were for the indica- 
tors 5_3 'Reliability caregivers' (3.69), 5_2 'Respectful 
treatment' (3.61), and 6_1 'Care plan and evaluation' 
(3.57). 

Although indicator 6_2 'Shared decision making' chan- 
ged the most between the t 0 and t lf the score at t x 
remained the lowest (2.98) of the interview question- 
naire. Indicators 6_3 'Information', 1_2 'Meals', and 6_6 
'Availability staff are ones that can make the most 
improvement in a future measurement (2.99, 3.04, 3.07, 
respectively), also because their 5 th and 95 th percentile 
range shows differences between the care homes (> 0.75 
range score). 

Indicator 2_4 ('Experience of safety') based on ques- 
tionnaires for representatives/family members barely 
changed between t 0 and t x . The score on indicator 6_2 
'Shared decision' increased considerably, but is still one 
of the lowest indicator scores. Indicators that scored 
high at t 0 and remained high at ti were indicators 2_3 
'Housing and privacy', 6_1 'Care plan and evaluation', 
7_11 'Physical restrains', and 1_2 'Meals'. 

Overall, the indicator scores of the questionnaire for 
assisted-living clients showed only minor changes (0.01- 
0.09). An exception to this was the indicator 6_2 
('Shared decision making'), which changed the most 
(0.25) but still had one of the lowest scores (2.94). Indi- 
cators scoring low at t 0 and remaining low were the 
indicators 3_1 'Daily activities and participation' (2.84) 
and 6_5 'Coherence in care' (3.08). 



Discussion 

This study investigated patients' experiences of the qual- 
ity of long-term care among the elderly. More specifi- 
cally, we looked at scores over time to evaluate whether 
indicator scores of nursing homes, residential care facil- 
ities and homecare providers, had improved and 
whether healthcare organizations that performed (much) 
below average improved more than organizations that 
performed on average and (much) above average. 

Our first hypothesis was that publishing the CQI sur- 
vey results of the first measurement (t 0 ) would trigger 
quality improvement activities which would lead to 
improved performance at the second measurement (t x ). 
Our results confirm that most indicator scores improved 
at Performance improved the most with respect to 
topics on the quality domain 'Quality of care organiza- 
tion'. Indicators belonging to this quality domain are 
care plan, shared decision making, and coherence in 
care. None of the indicator scores based on interviews 
with residents showed a decrease. The indicators based 
on mail questionnaires for representatives/assisted-living 
clients showed a decline of scores for three indicators. 
In addition, indicators that showed more dispersion of 
the 90% observed range at the first measurement also 
showed more improvement at the second measurement. 
This suggests that indicators with scores having a large 
range can more easily show improvement. 

A preliminary Dutch report failed to show an 
improvement in scores [28]; this might be because the 
authors analysed a much smaller sample of organiza- 
tional units and used data on the client level whereas 
our study was based on overall average scores per indi- 
cator per organizational unit. Moreover, data in that 
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study missed the variance per indicator scores for com- 
parison over time; nevertheless, the calculated scores of 
both measurement points are based on the original cor- 
rected scores with the confidence intervals [28]. On the 
other hand, our findings are consistent with other inter- 
national reports, as well as with theories on quality 
management and quality improvement [10,13,14]. Stu- 
dies in the USA have shown that homes reorganized 
quality improvement programs and started new quality- 
assurance programs in response to public performance 
scores [10,12,29]. Although there are no Dutch reports 
on specific actions taken by nursing homes, residential 
care facilities or homecare organizations to improve 
their scores, one qualitative study indicated that organi- 
zations start various quality improvement activities in 
response to the CQI results [30]. 

When examining specific changes in indicator scores, 
we see that the indicator 'Care plan and evaluation 
improved. One explanation for this is the modification 
made in the method of data collection of the interview 
protocol. Explaining the term 'care plan' was not 
allowed at t 0 whereas at ti the interviewers explained 
that 'care plan could be the 'green folder, 'red folder, or 
'care-living plan ', or whatever the care plan was called 
in that specific organizational unit. Another explanation 
for the improvement of indicator scores is involvement 
of the health insurer. If insurers indicate that a certain 
quality aspect is important and reward higher scores, 
organizations will probably improve more on those 
quality aspects. For the care plan, health insurers also 
have their own teams that perform administrative con- 
trols and trigger improvement in this area [30]. The 
change scores of some indicators of the mail question- 
naires for representatives and assisted-living clients are 
close to zero or even slightly negative. This could imply 
that changes in these areas are difficult to accomplish 
within a two-year period. 

The second hypothesis stated that nursing homes, 
residential care facilities, and homecare providers with 
a substandard performance at t 0 will show more 
improvement than those organizations whose perfor- 
mance was already relatively good. The overall change 
scores for the five performance groups showed that 
healthcare organizations in group 1 improved more 
than organizations in group 5. This relationship was 
stronger for the indicators based on the interview 
questionnaire with residents than for the indicators 
based on the mail questionnaires. For the mail ques- 
tionnaire for representatives, performance groups 3, 4 
and 5 did not differ significantly from each other 
regarding changes over time. These results are similar 
to those of Mukabel et al. who also found that care 
homes with poor quality scores were more likely to act 
on performance scores compared with those with 



better scores [10]. This might be related to the sense 
of urgency that managers/professionals in poor per- 
forming organizations may experience. For instance, 
Baier et al. found that homes with ambitious targets 
improve more than homes with less-ambitious targets 
[30]. More research is needed to elucidate the 
responses of nursing home staff/board members to the 
CQI results in terms of strategic orientation [31]. In 
the future, pay-for-performance may be a mechanism 
through which quality improvement can be achieved. 
In 2010, some healthcare purchasers have started to 
use CQI scores along with a number of other criteria 
to determine prices in long-term care. However, for 
individual care providers the financial consequences of 
performing low on CQI indicators, or of not publishing 
their results on the Internet were extremely limited. 

Limitations 

The present study has several limitations. 

First, the response categories in the interview protocol 
and questionnaires range from T never' to '4' always. 
This means that there is a ceiling effect when the fre- 
quency of the quality criteria is consistently good (score 
is 4). Some indicators reached this ceiling at t 0 , implying 
that there was little opportunity for improvement. For 
example, at ti the indicator 'Experience of safety' of the 
interview questionnaire had a score of 3.78 at ti and 
indicator 'Housing and privacy had a score of 3.76, indi- 
cating that the experiences of residents were very posi- 
tive and that poorer performing organizational units had 
more room for improvement. This ceiling effect was 
more profound for the interview questionnaire than for 
the mail questionnaires. This may be due to a mode 
effect, because (telephone) interviewees are reported to 
respond more positively than mail respondents [32]. 

Finally, the indicator score represents a case-mix cor- 
rected score per healthcare organization based on data 
on the level of the individual client. For the present 
study, we only had access to the average indicator scores 
per organizational unit. Thus, we could analyse 
improvement over time on the level of the organiza- 
tional unit, looking at different groups of organizations 
and aggregated indicator scores. However, because data 
at the client level were not provided by the national 
databank, it was not possible examine the variance of 
the indicator scores on the client level. Future studies 
should explore changes over time using data on the cli- 
ent level nested in organizational units. To analyse 
changes over time per organizational unit, multilevel 
analysis is necessary with: i) patients, ii) nested in orga- 
nizations, and iii) available per year. Use of this data 
processing method will allow to determine whether sin- 
gle organizational units have significantly improved over 
time. 
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Finally, we analysed the differences in performance 
between t 0 and t^ Although we assume that organiza- 
tional units have engaged in quality improvement strate- 
gies between these two points, we had no details on 
such quality improvement activities. Also, we have no 
data on the differences in the characteristics of respon- 
dents the between the two measurement point. It seems 
unlikely that the changes in performance between tO 
and tl were influenced by differences in case-mix as 
only data was used that were corrected for casemix. 

Conclusions 

Comparison of long-term care indicator scores over 
time revealed that indicator scores did improve. Nursing 
homes, residential care facilities, and homecare provi- 
ders with substandard performance on the first mea- 
surement showed more improvement than those 
organizations whose performance was already relatively 
good. 

Additional material 



Additional file 1: Figures of the mail questionnaire for 
representatives and assistend-living clients. Per indicator, the average 
score of the first measurement (t 0 ) and the change score (t] - 1 0 ) of the 
two different mail questionnaires are displayed. 



Acknowledgements 

The authors thank all the organizations that made their Annual Report 
publically available via http://www.jaarverslagenzorg.nl. 

Author details 

^RANZO, Scientific Centre for care and welfare, Faculty of Social and 
Behavioral Sciences, Tilburg University, PO Box 90153, 5000 LE Tilburg, the 
Netherlands. 2 Centre for Consumer Experience in Healthcare, P.O. Box 1568, 
3500 BN Utrecht, the Netherlands. 3 NIVEL, Netherlands Institute for Health 
Services Research, PO Box 1568, 3500 BN Utrecht, the Netherlands. 4 IQ 
Healthcare, Scientific Institute for Quality of Healthcare, Radboud University 
Nijmegen Medical Centre, PO Box 9101, 114, 6500 HB Nijmegen, the 
Netherlands. 

Authors' contributions 

MZ analysed and interpreted the data, and wrote a draft manuscript. DD, 
DdB, KL, and GW made critical revisions to the manuscript. All authors have 
read and approved the final manuscript. 

Competing interests 

The authors declare that they have no competing interests. 

Received: 22 December 2010 Accepted: 31 January 2012 
Published: 31 January 2012 

References 

1. Van Nie NC, Hollands LJ, Hamers JP: Reporting quality of nursing home 
care by an internet report card. Patient Educ Couns 2010, 78(3):337-343. 

2. Du Moulin MF, van Haastregt JC, Hamers JP: Monitoring quality of care in 
nursing homes and making information available for the general public: 
state of the art. Patient Educ Couns 2010, 78(3):288-296. 

3. Grabowski DC: Nursing home report cards. Med Care 2010, 48(1 0):859-861 . 

4. ActiZ: Quality Framework Responsible Care. The vision documents 
Towards standards for Responsible Care' and 'Standards for Responsible 



Home Care' made operational via a set of indicators and a control 
model for long-term and/or complex care. Utrecht: ActiZ; 2007. 

5. Donabedian A: Explorations in quality assessment and monitoring. Ml: 
Health Administration Press; 1980. 

6. Delnoij DM: Measuring patient experiences in Europe: what can we learn 
from the experiences in the USA and England? Eur J Public Health 2009, 
19(4)354-356. 

7. Delnoij D, Rademakers J, Groenewegen P: The Dutch Consumer Quality 
Index: an example of stakeholder involvement in indicator 
development. BMC Health Serv Res 2010, 10(88). 

8. Staatssecretaris lanceert kwaliteitsgegevens op kiesBeter.nl. [http://www. 
zorgvoorbeter.nl/onderwerpen/over/verantwoorde-zorg/nieuwsberichten/ 
staatssecretaris-lanceert-kwaliteitsgegevens-1200-locaties-op-kiesbeternl]. 

9. Triemstra M, Winters SW, Kool RB, Wiegers TA: Measuring client 
experiences in long-term care in the Netherlands: a pilot study with the 
Consumer Quality Index Long-term Care. BMC Health Serv Res 2010, 
10(95). 

10. Mukamel DB, Spector WD, Zinn JS, Huang L, Weimer DL, Dozier A: Nursing 
homes' response to the nursing home compare report card. J Gerontol B 
Psychol Sci Soc Sci 2007, 62(4):S2 18-225. 

11. Werner RMK, Stuart RT, Norton E, Park EC, J : The inpact of public 
reporting on quality of post-acute care. The Gerontologist 2009, 
49(6):793-802, Washington, DC. Academic Health 2008: In: Mukamel DB, et 
al. Is There Evidence of Cream Skimming Among Nursing Homes Following 
the Publication of the Nursing Home Compare Report Card?. 

12. Castle NG: Nursing home administrators' opinions of the nursing home 
compare web site. Gerontologist 2005, 45(3):299-308. 

13. Fung CH, Lim YW, Mattke S, Damberg C, Shekelle PG: Systematic review: 
the evidence that publishing patient care performance data improves 
quality of care. Ann Intern Med 2008, 148(2):1 1 1-123. 

14. Hendriks M, Spreeuwenberg P, Rademakers J, Delnoij DM: Dutch 
healthcare reform: did it result in performance improvement of health 
plans? A comparison of consumer experiences over time. BMC Health 
Serv Res 2009, 9:167. 

15. Zichtbare Zorg W&T: Quality Framework Responsible care 
(Kwaliteitskader Verantwoorde Zorg). Den-Haag: Zichtbare Zorg W&T; 
2010. 

16. Zuidgeest M, Sixma H, Rademakers J: Measuring patients' experiences 
with rheumatic care: the consumer quality index rheumatoid arthritis. 
Rheumatol Int 2009, 30(2):1 59-1 67. 

17. Damman OC, Hendriks M, Triemstra AHM, Sixma HJ: CQ-index Breast Care. 
Quality of care in breast abnormalities from patiens' perspective [CQ- 
index Mammacare: meetinstrumentontwikkeling. Kwaliteit van zorg 
rondom een borstafwijking vanuit het perspectief van patienten]. 
Utrecht: NIVEL; 2007. 

18. de Boer D, Delnoij D, Rademakers J: Do patient experiences on priority 
aspects of health care predict their global rating of quality of care? A 
study in five patient groups. Health Expect 2010, 13(3):285-97, Epub 2010 
Jun 9. 

19. Wiegers TA, Stubbe JH, Triemstra M: Development of the CQ-index 
Nursing Homes and Homecare: quality of life for residents, 
representatives and clients (Ontwikkeling van een CQ-lndex voor 
verpleeg- en verzorgingshuizen en thuiszorg: kwaliteit van zorg volgens 
bewoners, vertegenwoordigers en clienten). Utrecht: NIVEL; 2007. 

20. Stubbe JH, Brouwer W, Delnoij DM: Patients' experiences with quality of 
hospital care: the Consumer Quality Index Cataract Questionnaire. BMC 
Ophthalmol 2007, 7:14. 

21. Stubbe JH, Gelsema T, Delnoij DM: The Consumer Quality Index Hip Knee 
Questionnaire measuring patients' experiences with quality of care after 
a total hip or knee arthroplasty. BMC Health Serv Res 2007, 7:60. 

22. Damman OC, Stubbe JH, Hendriks M, Arah OA, Spreeuwenberg P, 
Delnoij DM, Groenewegen PP: Using multilevel modeling to assess case- 
mix adjusters in consumer experience surveys in health care. Med Care 
2009, 47(4):496-503. 

23. Arling G, Lewis T, Kane RL, Mueller C, Flood S: Nursing Home Performance. 
Improving Quality Assessment through Multilevel Modeling: The Case of 
Nursing Home Compare. Health Research and Educational Trust 2007, 42:3, 
Part 1. 

24. Thomas N, Longford NT, Rolph JE: Emperical bayes methods for 
estimating hospital-specific mortality rates. Statistics in Med 1994, 
13:889-903. 



Zuidgeest et al. BMC Health Services Research 2012, 12:26 
http://www.biomedcentral.eom/1 472-6963/1 2/26 



Page 1 0 of 1 0 



25. Goldstein H, Healy MJR: The Graphical Presentation of a Collection of 
Means. J Roy Stat Soc Sta 1 995, 1 58:1 75-1 77. 

26. Boer de D, Damman OC, Delnoij DMJ: Measurement explaination of client 
indicators sector nursing, care and home care (Meetverantwoording 
clientgebonden indicatoren W&T). Utrecht: Centrum Klantervaring Zorg; 
2008. 

27. Kirk RE: Practical significance: a concept whose time has come. Educ 
Psychol Meas 1996, 56(5):746-759. 

28. Winters S, Deen E: Leads the measurement of client experiences to 
better elderly care? [Leidt de meting van clientervaringen tot betere 
ouderenzorg?]. Kwaliteit in Zorg 2009, 6:17-19. 

29. Mukamel DB, Weimer DL, Spector WD, Ladd H, Zinn JS: Publication of 
quality report cards and trends in reported quality measures in nursing 
homes. Health Serv Res 2008, 43 (4): 1 244-1 262. 

30. Zuidgeest M, Luijkx K, Westert G, Delnoij D: Directing on client 
experiences. Transparancy and quality promoting in residential care 
facilities and nursing homes (Sturen op clientervaringen. Transparantie 
in de verpleging, verzorging en thuiszorg). Tilburg: Tilburg University; 
2010. 

31. Zinn JS, Spector WD, Weimer DL, Mukamel DB: Strategic orientation and 
nursing home response to public reporting of quality measures: an 
application of the Miles and Snow typology. Health Serv Res 2008, 

43(2)598-615. 

32. De Vries H, Elliott MN, Hepner KA, Keller SD, Hays RD: Equivalence of mail 
and telephone responses to the CAHPS Hospital Survey. Health Serv Res 
2005, 40(6 Pt 2):21 20-21 39. 

Pre-publication history 

The pre-publication history for this paper can be accessed here: 
http://www.biomedcentral.eom/1 472-6963/1 2/26/prepub 



doi:1 0.1 1 86/1 472-6963-1 2-26 

Cite this article as: Zuidgeest et al:. Patients' experiences of the quality 
of long-term care among the elderly: comparing scores over time. BMC 

Health Services Research 2012 12:26. 



Submit your next manuscript to BioMed Central 
and take full advantage of: 

• Convenient online submission 

• Thorough peer review 

• No space constraints or color figure charges 

• Immediate publication on acceptance 

• Inclusion in PubMed, CAS, Scopus and Google Scholar 

• Research which is freely available for redistribution 

Submit your manuscript at (^\ RioMM i r pnt ^\ 

www.biomedcentral.com/submit Blomea ^ enirai 



